Immersive telepresence anywhere

ABSTRACT

A conferencing peripheral for use with a mobile device or laptop or desktop computer can include one or more of projectors, cameras, microphones, and speakers. Such a device can work with the mobile device to provide a higher quality conferencing experience than has been provided to date by projecting a substantially full size, high resolution, image of conference participants onto a screen or wall and by providing microphones, speakers, and sufficient audio processing to provide high fidelity audio as part of the conferencing experience. The peripheral may be configured to use the voice and/or data network of the mobile device or may include its own internal network interface.

BACKGROUND

As videoconferencing technology has improved, it has come to be morewidely deployed. One problem that has historically existed withvideoconferencing technology is that the technology interferes with theconferencing experience. Small screen sizes, low quality video, and lowquality audio were the norm and created barriers to personal interactionthat do not appear in face-to-face meetings. Immersive telepresence(“ITP”) systems, which employ a combination of large screens, highquality video, and high quality audio have been developed to overcomethese advantages.

One such system is described in co-pending patent application Ser. No.12/249,467, filed Oct. 10, 2008, entitled Integrated System forTelepresence Videoconferencing. Such systems include numerous featuresdesigned to enhance the user's interaction experience. For example,multiple cameras and codecs coupled with relatively large screens allowthe remote participants to appear very near their actual size. This,coupled with high quality video can give the feeling of a face-to-facemeeting. Many such systems also employ custom furniture that is verysimilar at the local and remote endpoints. This, too helps provide theillusion of a face-to-face meeting. Other features, such as highfidelity audio dedicated presentation screens, and the like alsocontribute to the enhanced conferencing experience delivered by ITPsystems.

However, one drawback of today's ITP systems is that they are veryexpensive. Such systems can often cost on the order of $1,000,000.Another disadvantage is that such systems also require complex systemset up, a time consuming process to install (as much as 3-6 months) andexpensive maintenance. Another disadvantage is that the conference roomsused for immersive telepresence conferencing require substantiallydedicated real estate (around 30×25 ft). As a result of these drawbacks,one commercial ITP system, Polycom's RPX—Real Presence Experience,provides all of the benefits outlined above, but is only available forabout 0.001% of videoconferencing users because of its cost, complexity,and lack of portability.

On the other hand, relatively simple, low cost, and portableconferencing solutions exist. Many of these solutions are starting toprovide high quality audio and video that rival the ITP systems. Anexample of such a system is Polycom's Real Presence Mobile application,which brings high quality videoconferencing to portable devices, such aslaptop computers, tablet computers, and smart phones. Polycom RealPresence Mobile can solve problems for people connecting andcollaborating face-to-face using HD audio, video and content sharingwith colleagues and partners from any one, any where. However, Polycom'sReal Presence Mobile solution cannot—by itself—provide an immersivetelepresence experience because of the very small screen size on mobileplatforms and because of the relatively poor eye contact such systemsallow. Rendering far end video on mobile platforms such as an iPhone ortablet computer allow a user to hear and see things at a remote site,but do not create the feeling of “being there” one would get from a trueITP system.

SUMMARY

A conferencing peripheral for use with a mobile device or laptop ordesktop computer can include one or more of projectors, cameras,microphones, and speakers. Such a device can work with the mobile deviceto provide a higher quality conferencing experience than has beenprovided to date by projecting a substantially full size, highresolution, image of conference participants onto a screen or wall andby providing microphones, speakers, and sufficient audio processing toprovide high fidelity audio as part of the conferencing experience. Theperipheral may be configured to use the voice and/or data network of themobile device or may include its own internal network interface.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the present invention will be more readilyunderstood from reading the following description and by reference tothe accompanying drawings, in which:

FIG. 1 illustrates one embodiment of an immersive telepresenceconferencing peripheral.

FIG. 2 illustrates further details of an immersive telepresenceconferencing peripheral.

FIG. 3 illustrates an immersive telepresence conferencing peripheralproviding a simulated multi-screen experience.

FIG. 4 illustrates an alternative embodiment of an immersivetelepresence conferencing peripheral.

FIG. 5 illustrates further details of the immersive telepresenceconferencing peripheral illustrated in FIG. 4.

FIG. 6 illustrates a schematic diagram of the internal electronics of animmersive telepresence conferencing peripheral.

DETAILED DESCRIPTION

Described herein are apparatus, systems, and methods for expanding aportable conferencing experience, such as that provided by Polycom'sReal Presence Mobile to provide an enhanced conferencing experience. Oneaspect of providing this experience is the size at which remoteparticipants appear. The inventors have determined that to have alife-like real presence experience, objects (such as remoteparticipants) need to be rendered at least 50% of their true size on avideo display located at a reasonable meeting/social distance, whichinventors have determined to be about 5 to 10 feet. Obviously thiscannot be accomplished with the displays of smart phones, tablets,portable computers, or even—in many cases—desktop computers. However,recent developments in so-called “pico projection technology” (a/k/ahandheld, pocket, or mobile projectors) can facilitate a portable ITPexperience. By providing a pico projector that can be used inconjunction with a high-quality videoconferencing application on aportable device (such as a mobile phone, tablet computer, laptopcomputer, or the like) a user can project the far-side participants of acall on a wall or other suitable surface to achieve aforementioned sizeand distance parameters necessary for an ITP experience.

The implementation details of various pico projection technologies arenot pertinent to the inventive concepts described herein. Varioussolutions have been developed based on digital light processing (DLP),beam-steering, and liquid crystal on silicon (“LCoS”) technologies. Itis envisioned that any of these or other pico projection technologiescould be used in implementing the apparatus, methods, techniques, andsystems described herein. It should be noted that one additionaladvantage of these various technologies is that their relatively lowcost (as compared to traditional ITP systems) will improve theavailability of immersive telepresence solutions.

Illustrated in FIG. 1 is one example of a conferencing peripheral 101that can be used to deliver a low-cost, portable, immersive telepresenceexperience. Conferencing peripheral 101 is illustrated as a triangularperipheral designed to connect with a smartphone 102. The conferencingperipheral can be disposed, for example, on a tabletop (or othersuitable surface) 103 and oriented to project an image 104 of one ormore remote participants on a wall (or other suitable surface) 105.Further aspects of conferencing peripheral 101 will now be discussedwith reference to FIG. 2, which shows enlarged views of the conferencingperipheral 101. View 101 a is a perspective “front” view of theperipheral, while view 101 b is a “rear” view of the same device. Itshould be noted that the terms “front” and “rear” in this context aresomewhat arbitrary and are selected so that the “front” is facing theuser with the “rear” pointed towards the projection surface in normaloperation.

In the illustrated embodiment, peripheral 101 can be built from asubstantially triangular frame. The frame can be constructed from moldedplastic in one or more interlocking segments. The frame can also behollow, allowing for various electronic components discussed below to beenclosed therein. As shown in FIG. 2, rear view 102 b, the device caninclude a pico projector 201 located at an upper vertex of theconferencing peripheral. The projector can be controlled to project avideo image 104 on the screen/wall 105. In some embodiments theprojected image 104 can scale in resolution and size, for example from480p to 1920p and 36 to 86 inches or more, depending on how far theprojecting surface (wall) 105 is from the location of the unit. Distanceto the wall 105 (projecting surface) can be calculated by usingIR/ultrasonic reflection sensors on the unit (not shown). This can allowthe image to be auto scaled as appropriate to create an ITP experience.

At one or more locations around the frame, the molded plastic making upthe frame can be constructed with a plurality of openings, forming a“mesh grille” 202 that allows one or more microphones and one or morespeakers to be disposed within the frame. Alternatively, one or moreopenings in the frame may be covered with a fabric mesh to achieve thesame effect.

In some embodiments, it may be desirable to include a plurality ofmicrophones within the frame. This can allow the device to employ beamforming or other positional audio techniques that enhance theconferencing experience. A discussion of the details of implementingsuch an arrangement can be found in U.S. Pat. Nos. 6,593,956 and6,912,178. Such data can be used for a variety of activities, includingcamera framing (discussed further below). Details of a positional audiobased camera framing are described in U.S. Pat. No. 8,248,448.Additionally, multiple microphones could be used as part of an enhancedecho cancellation scheme, noise cancellation or various otherconferencing applications for which the use of multiple microphones isnecessary or advantageous.

Similarly, it may be desirable to include multiple speakers behind thespeaker grilles. These multiple speakers can be used in many ways toenhance the conferencing experience. For example, they can be used toprovide stereo audio that corresponds to the location of a speaker, asdescribed further below.

Conferencing peripheral 101 can also include one or more cameras 205. Tofacilitate the high quality video traditionally associated withtelepresence conferencing, camera 205 is preferably a high definitioncamera. Additionally, further camera elements could also be included andaimed so as to provide a variety of conference experiences. For example,cameras could be located at each of the remaining vertices of theconferencing peripheral 101 and aimed to capture participants sitting toeither side of the primary participant, further enhancing thetelepresence experience. Alternatively, one or more additional camerascould be configured as document or exhibit cameras, allowing the localparticipant to show a document or other item to the far-end participantswithout disrupting display of the local participant's image.Additionally, the cameras can be enabled with electronic pan, tilt, andzoom functionality allowing for optimal framing of the image transmittedto the far end.

Further components of the conferencing peripheral 101 include one ormore conferencing device interfaces. The illustrated embodiment includestwo such interfaces. A first interface 203 a can be disposed inside theperiphery of the triangular body of the peripheral 101 and can beconfigured to receive a smart phone, personal digital assistant, tabletcomputer, or similar device 204. Collectively these devices, asdistinguished from laptop or desktop computers and the like, will bedescribed as mobile devices. In some embodiments conferencing softwarerunning on the device (such as Polycom's Real Presence Mobile) can beused to control the various features and operation of the device.Interface 203 a is illustrated as a docking cradle type interface, butother interfaces, including wireless interfaces are also contemplated.

Depending on the capabilities of the mobile device 204, its owncamera(s), microphone(s), speaker(s) and/or screen(s) can be used inaddition to or in place of camera(s), microphone(s), speaker(s) and/orprojector(s) of the peripheral. For example, it might be desirable touse the microphone(s) of the mobile device 204 as an adjunct orsubstitute for the microphones in the peripheral 101. The same goes forspeakers and cameras. For example, a back-side camera 206 of the mobiledevice 204 could serve as a very effective document camera for showing ashared paper document to the remote participants. This is particularlyso for mobile devices that include a high resolution back side camera.In an embodiment in which certain functionality is provided by themobile device (or notebook/desktop computer), such as camera(s),microphone(s) or speaker(s), such components could be omitted from theconferencing peripheral 101.

Similarly, a display screen of the mobile device 204 could be used as anadjunct to the images projected by projector 201. For example, it mightbe useful to display content being presented by a remote participantusing the mobile device screen so that both the remote participant andthe content are visible at the same time. Alternatively, the mobiledevice screen and associated inputs (e.g., a touch screen or wirelessdevice) could be used by the local participant to manage content beingpresented at the far end. In yet another alternative, the display screenof the mobile device could be used to display one (or more) participantsin a multi point call, while the projected image(s) are used to displayother participants.

In some embodiments, the networking capabilities of mobile device 204,for example a cellular voice or data network, 802.11 wireless network(a/k/a “Wi-Fi”), can be used to complete the call. Alternatively, theperipheral 101 could include a separate wired or wireless networkinterface. This could either be its own separate interface, or part ofan additional interface 203 b used for connecting to a notebook ordesktop computer or the like. This additional interface could be, forexample, a universal serial bus (USB) or IEEE 1395 (a/k/a “Firewire”)interface, a wired or wireless Ethernet interface, or other suitableinterface. When used in conjunction with a notebook or desktop computerand alternate interface 203 b, the device could function insubstantially the same manner, with software on the computer serving asthe controller for the unit. As with mobile device 204, the I/O devicesof the computer (camera(s), speaker(s), microphone(s), display(s),network interface(s) and the like) could also be used as replacementsfor or adjuncts to such devices integral with the conferencingperipheral 101.

Further illustrated in FIG. 2, conferencing peripheral can include anadjustable stand 207. This adjustable stand can be used to adjust theangle of the device with respect to the surface 103 on which it rests,thereby adjusting the angle of projection as well as the angle of thecameras integrated with the peripheral and as part of mobile device 204.Not shown is a power interface, which can take a variety of forms,including internal batteries or a power adapter for connecting to ACmains power.

Illustrated in FIG. 3 is an alternative embodiment in which multipleprojectors are used create the experience of a multi-screen telepresencesystem. In the illustrated example three projectors (not shown), areused to generate three display images 104 a, 104 b, and 104 c. In oneembodiment, the individual projectors can be located in vertices of thetriangle (left, right and top) and can be used to show left, right andmiddle screens (images) on wall 105. Such a system need not be limitedto three screens and can be extended to have any desired number ofscreens. Additionally, the projectors can be located at any convenientlocation depending on device form factor. There is no requirement thatthey be located at vertices of the triangle.

Illustrated in FIG. 4 is an alternative to the embodiment describedabove. A portable conferencing unit 401 can be configured to work moreindependently of the connected mobile device or notebook/desktopcomputer platform. Such a device can include similar elements to thepreviously described embodiment (i.e., one or more projectors and also,potentially one or more of camera(s), projector(s), microphone(s),etc.). Portable conferencing unit 401 could also include additionalconferencing hardware and a network interface.

Portable conferencing unit 401 can be configured to stand on table 103substantially at eye level. This facilitates a realistic perspective forthe user, both because the camera will be able to capture the image ofthe local user at a natural angle and because the projected image of thefar end participants will not require excessive keystone correction. Aprojector on the opposite side of the unit (not shown) can project video104 from a connected laptop (or desktop) computer 402 or a tabletcomputer 403 (or other mobile device) onto the wall 105. In oneembodiment, microphones and/or speakers can be included in the main bodyor can be disposed in recesses in the tripod legs. As illustrated inFIG. 5, portable conferencing unit 401 can be constructed with foldingtripod legs that can fold up against the device for transportation andstorage.

In any of the illustrated embodiments, the unit/stand (with camera) canadjust the size and height of the projected image to facilitate bettereye contact of the person presenting. As noted above, it may bedesirable that the size of the participants be at least 50% of actualsize and projected on a surface about 5 to 10 feet away. It may also bedesirable to project the remote participants at such a height on thewall that the position of the eyes of the remote participants isapproximately collinear with the eyes of the local participant and thecamera of the conferencing peripheral or other camera used to capturethe image of the local participant. Eye position of both the local andremote participants can be determined by a face recognition algorithmapplied to the local and remote video streams. Approximate collinearityof the eyes of the local and remote participants and the local imagesensor can be determined on a case-by-case basis. In general, it hasbeen found that when an angle formed by the local participant's eyes,the remote participant's eyes, and the local image sensor is less than10 degrees, and preferably less than about 5 degrees, a sufficientperception of direct eye contact is created.

Illustrated in FIG. 6 is a block diagram schematically depicting thevarious components of a conferencing peripheral 601 according to thevarious embodiments described herein. Conferencing peripheral can becommunicatively coupled to a mobile device, laptop, or desktop computer602 (hereinafter portable device 602) via a conferencing deviceinterface 603. As noted above, the conferencing device interface may bea wired connection such as a serial link. Alternatively, it could be awireless connection. Conferencing peripheral 601 is controlledinternally by processor 604; however, the system can be designed so thatmuch of the processing takes place on mobile device 602 under thedirection of program code stored in a memory and executed by a processorof the mobile device. Depending on the details of implementation,processor 604 may be a relatively powerful and fully featuredmicroprocessor or DSP or may be a relatively simple processor that isbasically providing an interface to the various components ofconferencing peripheral 601. In such embodiments, peripheral interface603 could be separate or could be integrated with processor 604.

Processor 604 communicates with one or more additional components,including projector 605, camera 606, microphone 607, and speaker 608.Multiples of each of these devices may be supplied. Alternatively, someof these devices may be omitted depending on the particular embodiment.As depicted in FIG. 6, each of these components communicates with theprocessor via a common bus. However, depending on the details, they mayrequire their own additional interface circuitry (not shown) or they maycommunicate directly with the portable device 602 via the conferencingdevice interface. Processor 604 can optionally communicate with a memorystoring computer instructions for execution by processor 604.Additionally, with modern system on a chip (SoC) technology one or moreof conferencing device interface 603, network interface 609, and theinterface circuitry for connection with projector 605, camera 606,microphone 607, and/or speaker 608 could all be integrated withprocessor 604.

Peripheral 601 could be configured as a completely free-standingconferencing device, in which case the portable device 602 would not berequired for conferencing functionality. In that case conferencingdevice interface 603 might be omitted or might be provided merely as away for supplemental use of portable device 602, such as providingconference content. In such embodiments, a network interface 609 mightbe provided to facilitate connection to remote endpoints. Even inembodiments in which mobile device 602 and its network connection areused for conferencing, it might be desirable to provide an auxiliarynetwork interface to enable other functionality.

The above description is illustrative and not restrictive. Manyvariations of the invention will become apparent to those skilled in theart upon review of this disclosure. The scope of the invention shouldtherefore be determined not with reference to the above description, butinstead with reference to the appended claims along with their fullscope of equivalents.

What is claimed is:
 1. A portable immersive telepresence conferencingperipheral comprising: a frame incorporating one or more cameras, picoprojectors, speakers, and microphones, a power interface, and at leastone conferencing device interface configured to operatively couple theconferencing peripheral to a portable device running conferencingsoftware; wherein the conferencing peripheral is controllable by theconferencing software to: operate the one or more cameras andmicrophones to capture audio and video of one or more near endconference participants for communication to one or more far endconference participants via a media conference established by theconferencing software on the portable device; operate the one or morespeakers to deliver audio of one or more far end conference participantsreceived via the media conference; and operate the one or more picoprojectors to display images of the one or more far end conferenceparticipants received via the media conference such that the one or morefar end conference participants are rendered at substantially true sizeon a display surface located at a reasonable social distance from theone or more near end conference participants.
 2. The portable immersivetelepresence conferencing peripheral of claim 1 wherein the frame is ahollow, molded plastic frame, and the one or more cameras, picoprojectors, speakers, and microphones are disposed within the frame. 3.The portable immersive telepresence conferencing peripheral of claim 2wherein the frame is substantially triangular and wherein the one ormore pico projectors are disposed at one or more vertices of thesubstantially triangular frame on one side of the frame and wherein theone or more cameras are disposed at one or more vertices of thesubstantially triangular frame on an opposite side of the frame.
 4. Theportable immersive telepresence conferencing peripheral of claim 1wherein substantially true size is at least 50% of true size and whereinthe reasonable social distance is between 5 and 10 feet.
 5. The portableimmersive telepresence conferencing peripheral of claim 1 furthercomprising one or more infra red or ultrasonic sensors operable todetermine a distance from the one or more pico projectors to the displaysurface whereby the conferencing software can automatically scale thedisplay of the one or more far end conference participants.
 6. Theportable immersive telepresence conferencing peripheral of claim 1wherein the one or more microphones comprise a plurality of microphonesoperable to provide one or more functions selected from the groupconsisting of: beam forming, positional audio, camera framing, echocancellation, and noise cancellation.
 7. The portable immersivetelepresence conferencing peripheral of claim 1 wherein the one or morespeakers comprise a plurality of speakers operable to provide stereoaudio corresponding to a projected location of one or more far endconference participants.
 8. The portable immersive telepresenceconferencing peripheral of claim 1 wherein the one or more camerascomprise a plurality of cameras and wherein the plurality of cameras areoperable to image one or more items selected from the group consistingof: conference participants, documents, and exhibits.
 9. The portableimmersive telepresence conferencing peripheral of claim 1 wherein theone or more cameras are electronic pan-tilt-zoom cameras.
 10. Theportable immersive telepresence conferencing peripheral of claim 1wherein the at least one conferencing device interface is a dockingcradle for a mobile device.
 11. The portable immersive telepresenceconferencing peripheral of claim 1 further comprising a networkinterface, wherein the media conference may be established using eitherthe network interface of the conferencing peripheral or a networkinterface of the portable device.
 12. The portable immersivetelepresence conferencing peripheral of claim 1 further comprising anadjustable stand.
 13. A portable immersive telepresence conferencingperipheral comprising: at least one pico projector; a power interface;and a conferencing device interface configured to operatively couple theconferencing peripheral to a portable device running conferencingsoftware; wherein the conferencing peripheral is controllable by theconferencing software to operate the at least one pico projector todisplay an image of at least one far end conference participant receivedvia a media conference established by the portable device such that theat least one far end conference participant is rendered at least 50% oftrue size on a display surface.
 14. The portable immersive telepresenceconferencing peripheral of claim 13 wherein the conferencing peripheralincludes at least one camera disposed on an opposite side of theconferencing peripheral from the at least one pico projector and whereinthe conferencing peripheral is adapted to stand on a table atsubstantially eye level.
 15. The portable immersive telepresenceconferencing peripheral of claim 14 wherein the conferencing peripheralis adapted to stand on the table at substantially eye level via anadjustable folding tripod.
 16. The portable immersive telepresenceconferencing peripheral of claim 15 wherein legs of the adjustablefolding tripod are configured to be folded against a body of theconferencing peripheral for storage.
 17. The portable immersivetelepresence conferencing peripheral of claim 16 wherein legs of thetripod incorporate at least one microphone or at least one speaker. 18.The portable immersive telepresence conferencing peripheral of claim 15wherein the conferencing peripheral is configured to display the imageof the at least one far end conference participant such that eyes of theat least one far end conference participant are approximately collinearwith eyes of a local participant and a camera used to capture the imageof the local participant.
 19. The portable immersive telepresenceconferencing peripheral of claim 18 wherein eye position of both theparticipants is determined by a face recognition algorithm applied to alocal and a remote video stream.
 20. The portable immersive telepresenceconferencing peripheral of claim 15 further comprising at least onemicrophone and at least one speaker.
 21. The portable immersivetelepresence conferencing peripheral of claim 13 wherein theconferencing device interface is a wireless interface.
 22. A portableimmersive telepresence conferencing peripheral comprising: aconferencing device interface configured to operatively couple theconferencing peripheral to a portable device running conferencingsoftware; a processor operatively coupled to the conferencing deviceinterface for controlling operation of the conferencing peripheral; andat least one pico projector operatively coupled to the processor;wherein the pico projector is controlled by the processor to display animage of at least one far end conference participant received via amedia conference established by the portable device such that the atleast one far end conference participant is displayed at substantiallytrue size on a display surface.
 23. The portable immersive telepresenceconferencing peripheral of claim 22 wherein the at least one picoprojector comprises a plurality of projectors, each configured todisplay an image of at least one far end conference participant.
 24. Theportable immersive telepresence conferencing peripheral of claim 22further comprising at least one camera operatively coupled to theprocessor to facilitate capture of images of one or more near endconference participants for communication to the at least one far endconference participant via the media conference.
 25. The portableimmersive telepresence conferencing peripheral of claim 24 wherein theat least one camera comprises a plurality of cameras, wherein at leastone of the cameras is positioned to capture images of another near endconference participant, a document, or an exhibit.
 26. The portableimmersive telepresence conferencing peripheral of claim 22 furthercomprising at least one microphone operatively coupled to the processorto facilitate capture of audio of one or more near end conferenceparticipants for communication to the at least one far end conferenceparticipant via the media conference.
 27. The portable immersivetelepresence conferencing peripheral of claim 26 wherein the at leastone microphone comprises a plurality of microphones operable to provideone or more functions selected from the group consisting of: beamforming, positional audio, camera framing, echo cancellation, and noisecancellation.
 28. The portable immersive telepresence conferencingperipheral of claim 22 further comprising one or more speakersoperatively coupled to the processor to facilitate delivery of audio ofone or more far end conference participants received via the mediaconference.
 29. The portable immersive telepresence conferencingperipheral of claim 28 wherein the one or more speakers are operable toprovide stereo audio corresponding to a projected location of one ormore far end conferencing participants.
 30. The portable immersivetelepresence conferencing peripheral of claim 22 further comprising oneor more infra red or ultrasonic sensors operatively coupled to theprocessor to facilitate determination of a distance from the at leastone pico projectors to the display surface and automatic scaling of thedisplayed image of the one or more far end conference participants. 31.The portable immersive telepresence conferencing peripheral of claim 22wherein the conferencing device interface and the processor are a singlesystem on a chip.
 32. The portable immersive telepresence conferencingperipheral of claim 22 wherein the conferencing device interface is awireless interface.
 33. The portable immersive telepresence conferencingperipheral of claim 22 wherein the conferencing device interface is adock for a mobile device.
 34. The portable immersive telepresenceconferencing peripheral of claim 22 further comprising a networkinterface for establishing another media conference independently of theportable device.